Parameter-Efficient Fine-Tuning Method for Task-Oriented Dialogue Systems

The use of Transformer-based pre-trained language models has become prevalent in enhancing the performance task-oriented dialogue systems. These models, which are on large text data to grasp syntax and semantics, fine-tune entire parameter set according a specific task. However, as scale model increases, several challenges arise during fine-tuning process. For example, training time escalates grows, since complete needs be trained. Furthermore, additional storage space is required accommodate larger size. To address these challenges, we propose new system called PEFTTOD. Our proposal leverages method Parameter-Efficient Fine-Tuning (PEFT), incorporates an Adapter Layer prefix tuning into model. It significantly reduces overall count used efficiently transfers knowledge. We evaluated PEFTTOD Multi-WOZ 2.0 dataset, benchmark dataset commonly Compared traditional method, utilizes only about 4% parameters for training, resulting improvement combined score compared existing T5-based baseline. Moreover, achieved efficiency gain by reducing 20% saving up 95% space.